SPECIFICATION 



TO ALL WHOM IT MAY CONCERN: 

BE IT KNOWN THAT WE, TAKASHI KITAGUCHI, a 
citizen of Japan residing at Kanagawa, Japan and 
NORIHIKO MURATA, a citizen of Japan residing at 
Kanagawa, Japan have invented certain new and useful 
improvements in 

SHAPE MEASUREMENT SYSTEM 
of which the following is a specification:- 



BACKGROUND OF THE INVENTION 

1 . Field of the Invention 

The present invention generally relates to a 
computer inputting technology, a human interface 
technology, and a sight processing technology, and, in 
particular, to a technology of measuring a three- 
dimensional shape of an object. 

2 . Description of the Related Art 
Conventionally, to measure a three-dimensional 

shape of an object (subject) has been demanded in 
various fields such as survey of land/building, 
production of CAD model, display of products/goods in 
on-line shopping, and so forth. As one of such a three- 
dimensional shape measuring methods, there is a method 
by which a pattern light is irradiated/projected onto an 
object, a picture of distortion occurring in the pattern 
on the object is taken, and, thus, shape measurement is 
performed . 

Specifically, Japanese Laid-Open Patent 
Application No. 5-149727 discloses to take a picture of 
a subject by a camera which can move freely on a moving 
apparatus on which the camera is mounted, and thereby, 
to measure a shape of the subject even when the subject 
has a surface having high reflectance. Further, 
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Japanese Laid-Open Patent Application No. 10-79029 
discloses to divide each of a plurality of sets of image 
data taken by a plurality of cameras into small regions, 
to combine three-dimensional information obtained from 
5 the respective small regions, and, thus, to detect 
three-dimensional information of the subject. 

However, the above-mentioned technique 
requires the moving apparatus or plurality of cameras. 
Thereby, the apparatus becomes larger, and cannot be 

10 carried easily. Thus, the apparatus is not convenient. 
Furthermore, high manufacturing costs are needed for 
such apparatuses. Furthermore, depending on the actual 
arrangement of the moving apparatus or plurality of 
cameras, characteristics of a subject for which the 

15 shape thereof can be properly measured may be limited. 

SUMMARY OF THE INVENTION 

The present invention has been devised in 
order to solve these problems, and, an object of the 

20 present invention is to achieve measurement of three- 
dimensional shape at high accuracy which can be used in 
various conditions, and, to provide a shape measurement 
system and a picture taking device which are as a whole 
miniaturized and inexpensive, the shape measurement 

2 5 method, and a recording medium in which a softwar4e 
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program for performing the shape measurements method is 
recorded . 

A shape measurement system according to the 
present invention for measuring a three-dimensional 
5 shape of a subject, comprises: 

a picture taking part taking a picture of the 

subj ect ; 

a projecting part applying light having a 
predetermined pattern onto the subject ; 

10 a picture taking position specifying part 

detecting a position at which the picture taking part 
takes the picture of the subject, and generating 
position information specifying the position; 

a three-dimensional coordinate calculating 

15 part calculating a three-dimensional coordinate of each 
point of the subject in accordance with the position 
information and an image taken at the position specified 
by the position information; and 

a three-dimensional shape composing part 

20 expressing the each point by a coordinate in a single 
coordinate system and generating a composite image in 
accordance with at least two three-dimensional 
coordinates calculated for the each point by the three- 
dimensional coordinate calculating part based on 

25 respective images obtained as a result of a picture of 
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the subject on the light having the predetermined 
pattern is applied being taken from at least two 
different positions . 

Thereby, by three-dimensional coordinates 
5 calculated in accordance with position information and 
images obtained through taking pictures of a subject, a 
three-dimensional shape of the subject is obtained. 
Accordingly, it is possible to measure a three- 
dimensional shape at a high accuracy through a simple 
10 configuration. Thereby, it is possible to miniaturize 
the shape measurement system, simplify the shape 
measurement operation, and to provide the shape 
measurement system in which reliability is improved. 

The shape measurement system may further 

.15 comprise: 

a picture taking control part controlling 
operation timing of the picture taking part; 

a signal converting part converting an analog 
signal obtained by the picture taking part into a 
20 digital signal; and 

a storing part storing the digital signal, 
three-dimensional coordinate and composite image. 

The shape measurement system may further 
comprise an interpolation part performing interpolation 
25 processing on at least one of the image obtained by the 
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picture taking part and the composite image obtained by 
the three-dimensional shape composing part. Thereby, it 
is possible to obtain a three-dimensional shape at a 
higher accuracy. 
5 The shape measurement system may further 

comprise a three-dimensional image generating part 
generating a three-dimensional image of the subject in 
accordance with coordinates of the subject obtained by 
the three-dimensional coordinate calculating part and an 

10 image obtained when the light having the predetermined 
pattern is not applied onto the subject by the picture 
taking part. Thereby, it is possible to obtain a three- 
dimensional image in which even a pattern and so forth 
actually provided on the subject is reproduced. Thus, 

15 it is possible to obtain faithfully reproduced three- 
dimensional image of the subject. 

A shape measurement system according to 
another aspect of the present invention for measuring a 
three-dimensional shape of a subject, comprises: 

20 a plurality of picture taking parts with 

different optical centers taking pictures of the 
subj ect ; 

a projecting part applying light having a 
predetermined pattern onto the subject; 
25 a picture taking position specifying part 
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detecting positions at which the plurality of picture 
taking parts take the pictures of the subject, and 
generating position information specifying the 
respective positions; 
5 a three-dimensional coordinate calculating 

part calculating a three-dimensional coordinate of each 
point of the subject for each image in accordance with a 
plurality of images obtained as a result of pictures of 
the subject being taken by the plurality of picture 

10 taking parts, and the position information generated by 
the picture taking position specifying part; and 

a three-dimensional shape composing part 
expressing the each point by a coordinate in a single 
coordinate system and generating a composite image in 

15 accordance with the plurality of three-dimensional 

coordinates calculated for the each point by the three- 
dimensional coordinate calculating part. 

Thereby, it is possible to obtain a plurality 
of images through the plurality of picture taking parts 

20 simultaneously. Accordingly, by calculating three- 
dimensional coordinates from the plurality of images, 
respectively, and combining them, it is possible to 
achieve measurement of three-dimensional shape at a 
higher accuracy. Thereby, it is possible to further 

25 improve the reliability of the shape measurement system. 



The shape measurement system may further 

comprise : 

picture taking control parts each controlling 
operation timing of each of the plurality of picture 
5 taking parts ; 

signal converting parts converting an analog 
signals obtained by the picture taking parts into 
digital signals, respectively; and 

a storing part storing these digital signals, 
10 three-dimensional coordinates obtained from the three- 
dimensional coordinate calculating part and the 
composite image. 

A shape measurement system according to 
another aspect of the present invention for measuring a 
15 three-dimensional shape of a subject, comprises a 

picture taking device taking a picture of a subject and 
a computer. The picture taking device comprises: 

a projecting part applying light having a 
predetermined pattern onto the subject; and 
20 a picture taking position specifying part 

detecting a position at which the picture taking part 
takes the picture of the subject, and generating 
position information specifying the position; and the 
computer comprises: 
25 a three-dimensional coordinate calculating 
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part calculating a three-dimensional coordinate of each 
point of the subject in accordance with the position 
information and an image taken at the position specified 
by the position information; and 
5 a three-dimensional shape composing part 

expressing the each point by a coordinate in a single 
coordinate system and generating a composite image in 
accordance with at least two three-dimensional 
coordinates calculated for the each point by the three- 
10 dimensional coordinate calculating part based on 

respective images obtained as a result of picture of the 
subject on the light having the predetermined pattern is 
applied being taken from at least two different 
positions . 

15 Thereby, by three-dimensional coordinates 

calculated through the computer in accordance with 
position information and images obtained through taking 
pictures of a subject obtained through the picture 
taking device, a three-dimensional shape of the subject 

2 0 is obtained through the computer. Accordingly, it is 

possible to measure a three-dimensional shape at a high 
accuracy through a simple configuration. In this system, 
by utilizing the performance of the computer, it is 
possible to provide a three-dimensional image having a 

2 5 higher accuracy. 
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The computer may have an interpolation part 
which performs interpolation processing on the plurality 
of three-dimensional coordinates calculated by the 
three-dimensional coordinate calculating part. 
5 At least one of the projecting part and 

picture taking position specifying part may be 
controlled by the computer. 

A picture taking device according to the 
present invention comprises a picture taking part taking 
10 a picture of a subject, and further comprises: 

a projecting part applying light having a 
predetermined pattern onto the subject; 

a picture taking position specifying part 
detecting a position at which the picture taking part 
15 takes the picture of the subject, and generating 
position information specifying the position; and 

a storing part storing an image obtained as a 
result of a picture of the subject being taken by the 
picture taking part, and the position information. 
20 Thereby, it is possible to easily obtain data 

for measuring a three-dimensional shape of a subject. 

At least one of the projecting part and the 
picture taking position specifying part may be 
controlled by a control signal provided externally. 
25 The picture taking part may also take a 
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picture of the subject onto which the light having the 
predetermined pattern is not applied. Thereby, it is 
possible to generate a reproduced image of a subject 
including a pattern or the like provided on the subject. 
5 A shape measurement method of measuring a 

three-dimensional shape of a subject, comprising the 
steps of: 

a) applying light having a predetermined 
pattern onto the subject; 
10 b) detecting a position at which the step a) 

takes the picture of the subject, and generating 
position information specifying the position; 

c) calculating a three-dimensional coordinate 
of each point of the subject in accordance with the 

15 position information and an image taken at the position 
specified in the step b) ; and 

d) expressing the each point by a coordinate 
in a single coordinate system in accordance with at 
least two three-dimensional coordinates calculated for 

20 each point in the step c) based on respective images 

obtained as a result of picture of the subject on which 
the light having the predetermined pattern is applied 
being taken from at least two different positions in the 
step a) . 

25 Thereby, by three-dimensional coordinate 
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calculated in accordance with position information and 
image obtained through taking a picture of a subject, a 
three-dimensional shape of the subject is obtained. 
Accordingly, it is possible to easily measure a three- 
5 dimensional shape at a high accuracy. 

The method may further comprise the step of: 
e) generating a three-dimensional image of the 
subject in accordance with the coordinates of the 
subject in the coordinate system, and an image of the 

10 subject obtained in the step a) but when the light 
having the predetermined is not applied thereonto . 
Thereby, it is possible to generate a three-dimensional 
reproduced image which is faithful to the subject even 
for a pattern or the like provided on the subject. 

15 A shape measurement method of measuring a 

three-dimensional shape of a subject according to 
another aspect of the present invention, comprises the 
steps of: 

a) applying light having a predetermined 
20 pattern onto the subject; 

b) taking pictures of the subject through a 
plurality of picture taking parts having direction 
optical centers; 

c) detecting positions at which the plurality 
25 of picture taking parts take the pictures of the subject, 
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and generating position information specifying the 
positions, respectively ; 

d) calculating a three-dimensional coordinate 
of each point of the subject for each image in 
5 accordance with a plurality of images obtained as a 

result of pictures of the subject on which the light of 
the predetermined is applied being taken by the 
plurality of picture taking parts, and the position 
information generated in the step c) ; and 

10 e) expressing the each point by a coordinate 

in a single coordinate system in accordance with the 
three-dimensional coordinates of the each point 
calculated in the step d) . 

Thereby, it is possible to obtain a plurality 

15 of images through the plurality of picture taking parts 
simultaneously. Accordingly, by calculating three- 
dimensional coordinates from the plurality of images, 
respectively, and combining them, it is possible to 
achieve measurement of three-dimensional shape at a 

2 0 higher accuracy. 

A computer readable recording medium according 
to the present invention in which a program for 
measuring a three-dimensional shape of a subject through 
a compute is provided. The program causes the computer 

25 to: 
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calculate a three-dimensional coordinate of 
each point of the subject in accordance with an image 
obtained as a result of picture of the subject on which 
light having a predetermined pattern is applied being 
5 taken and position information specifying a position at 
which the picture of the subject is thus taken; and 

express the each point by a coordinate in a 
single coordinate system in accordance with at least two 
three-dimensional coordinates calculated for the each 
10 point based on respective images obtained as a result of 
picture of the subject on which the light having the 
predetermined pattern is applied being taken from at 
least two different positions. 

Thereby, by three-dimensional coordinate 
15 calculated in accordance with position information and 
image obtained through taking a picture of a subject 
through execution of a software program by a computer, a 
three-dimensional shape of the subject is obtained. 
Accordingly, it is possible to easily measure a three- 
20 dimensional shape at a high accuracy through the 
software program. 

The program may cause 

an acceleration sensor to generate the 
position information specifying the position with 
25 respect to a gravitation; and 
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a magneitic sensor to generate the position 
information specifying the position with respect to 
terrestrial magnetism. 

The program may cause an angular velocity 
5 sensor to detect a rotational angular velocity around 

each coordinate axis of the three-dimensional coordinate 
system. 

The program may further cause the computer to 
generate a three-dimensional image of the subject in 

10 accordance with the coordinates of the subject in the 
single coordinate system, and an image of the subject 
obtained through taking picture of the subject but on 
which the light having the predetermined is not applied, 
it is possible to generate a three-dimensional 

15 reproduced image which is faithful to the subject even 
for a pattern or the like provided on the subject. 

A computer readable recording medium according 
to another aspect of the present invention for measuring 
a three-dimensional shape of a subject through a 

2 0 computer is provided. The program causes the computer 
to : 

calculate a three-dimensional coordinate of 
each point of the subject for each image in accordance 
with a plurality of images obtained as a result of 
25 pictures of the subject on which the light of the 
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predetermined is applied being taken by the plurality of 
picture taking parts, and the position information 
specifying positions at which the pictures of the 
subject are taken by the plurality of picture taking 
5 parts ; and 

express the each point by a coordinate in a 
single coordinate system in accordance with the thus- 
calculated three-dimensional coordinates for the each 
point . 

10 Thereby, it is possible to obtain a plurality 

of images through the plurality of picture taking parts 
simultaneously through execution of a software program 
by a computer. Accordingly, by calculating three- 
dimensional coordinates from the plurality of images, 

15 respectively, and combining them, it is possible to 
achieve measurement of three-dimensional shape at a 
higher accuracy. 

Other objects and further features of the 
present invention will become more apparent from the 

20 following detailed description when read in conjunction 
with the accompanying drawings . 



BRIEF DESCRIPTION OF THE DRAWINGS 

FIG. 1 shows a block diagram of a shape 
25 measurement system in a first embodiment according to 
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the present invention; 

FIG . 2 shows an outline of a camera in which 
the shape measurement system shown in FIG. 1 is 
included ; 

5 FIG. 3 shows an operation flow chart which 

illustrates operation of the shape measurement system 
shown in FIG. 1; 

FIG. 4 illustrates picture taking operation of 
the shape measurement system shown in FIG. 1; 
10 FIG. 5 is a block diagram of an attitude 

detecting part shown in FIG. 1; 

FIG. 6 illustrates attitude angle calculating 
operation performed by the shape measurement system 
shown in FIG. 1; 
15 FIGS. 7 A through 7D and 8A through 8B 

illustrate three-dimensional coordinate composing 
operation performed by the shape measurement system 
shown in FIG. 1; 

FIG. 9 shows an example in which the shape 
20 measurement system shown in FIG. 1 is embodied by a 
camera and a personal computer ; 

FIG. 10 shows a block diagram of the camera 
and personal computer shown in FIG. 9; 

FIG. 11 shows another example in which the 
25 shape measurement system shown in FIG. 1 is embodied by 



-18- 



a camera and a personal computer; 

FIG. 12 shows an example in which the shape 
measurement system shown in the first embodiment of the 
present invention is embodied by a computer software 
5 executed by a personal computer; 

FIG. 13 shows a block diagram of a shape 
measurement system in a second embodiment according to 
the present invention; 

FIGS. 14A and 14B illustrate interpolation 
10 processing performed by the shape measurement system 
shown in FIG. 13; 

FIGS. 15A and 15B illustrate correction 
processing performed by the shape measurement system 
shown in FIG. 13; 
15 FIG. 16 shows an example in which the shape 

measurement system shown in FIG. 13 is embodied by a 
personal computer and a camera; and 

FIG. 17 shows a block diagram of a shape 
measurement system in a third embodiment according to 
20 the present invention. 

DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS 

Embodiments of the present invention will now 
be described in detail with reference to figures. The 
25 same reference numerals represent same or corresponding 
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parts/components . 

FIG. 1 shows a block diagram of a shape 
measurement system in a first embodiment of the present 
invention. As shown in FIG. 1, the shape measurement 
5 system 1 in the first embodiment includes an MPU (Micro 
Processing Unit) 3, an external terminal 5, a stop 
(aperture) mechanism 6, a correlated double sampling 
circuit (CDS) 7, a timing signal generating circuit 8, 
an IPP (Image Pre-Processor ) circuit 9, a lens 10, a 
10 picture taking device 11 such as a CCD, an A-D converter 
12, an image memory 13, a pattern light irradiating part 

14, a 3D (three-dimensional) position calculating part 

15, an attitude detecting part 16 and a three- 
dimensional position composing part 17. 

15 The CDS circuit 7 is connected to the picture 

taking device 11, and the A-D converter 12 is connected 
to the CDS circuit 7. Further, the IPP circuit 9 is 
connected to the A-D converter 12, and the image memory 
13 is connected to the IPP circuit 9. Furthermore, the 

20 3D position calculating part 15 and three-dimensional 
position composing part 17 are connected to the image 
memory 13. 

The picture taking device 11, CDS circuit 7 
and A-D converter 12 are connected to the timing signal 
25 generating circuit 8. The external terminal 5, stop 
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mechanism 6, timing signal generating circuit 8, IPP 
circuit 9, image memory 13, 3D position calculating part 
15, three-dimensional position composing part 17, 
pattern light irradiating part 14 and attitude detecting 
5 part 16 are connected to the MPU 3. 

In the above-described shape measurement 
system 1, when a picture of a subject (not shown in the 
figure) is taken, an image of the subject is formed on 
the picture taking device 11 through the lens 10 and 

10 stop mechanism 6. Then, an image signal thus obtained 
through the picture taking device 11 is sampled by the 
CDS circuit 7, and, then, is converted into a digital 
signal through the A-D converter 12. Operating timing 
in this occasion is controlled by a signal provided by 

15 the timing signal generating circuit 8. 

The thus-obtained digital image signal 
undergoes image processing such as aperture correction 
and/or data compressing through the IPP circuit 9, and 
then is stored in the image memory 13. In this occasion, 

20 it is also possible that the MPU 3 performs the various 
correction processing and/or compressing processing on 
the image signal. 

The pattern light irradiating part 14 
irradiates/projects pattern light having a slit pattern, 

25 a dot pattern, or tone gradation onto the subject. Then, 
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based on a picture/image taken from the subject on which 
the pattern light is irradiated/projected, the 3D 
position calculating part 15 calculates three- 
dimensional coordinates of points present on the surface 
5 of the subject. 

The attitude detecting part 16 detects 
attitude (picture taking position, picture taking angle 
and so forth) of the shape measurement system 1 at an 
occasion of taking a picture of the subject. The above- 

10 mentioned attitude is detected each time a picture is 
taken by the system 1 . Information such as the above- 
mentioned picture taking position, picture taking angle 
and so forth may be absolute one, or may be relative one 
with respect to a viewpoint. 

15 The three-dimensional position composing part 

17 combines a plurality of images obtained as a result 
of pictures of the subject being thus taken, in 
accordance with information concerning the corresponding 
to attitudes obtained through the attitude detecting 

20 part 16, and, thereby, obtains a three-dimensional shape 
of the subject. The respective circuits shown in FIG. 1 
are controlled by the MPU 3 directly or indirectly. 

The above-described shape measurement system 1 
may be formed as a form of a camera 2 as shown in FIG. 2 

25 as a result of being contained in a housing. A 
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carriable recording medium such as a floppy disk 172 may 
be loaded into the camera 2, and, thereby, the generated 
three-dimensional image can be stored in the carriable 
recording medium by the camera 2 . 
5 Operation performed by the above-mentioned 

shape measurement system 1 will now be described in 
detail with reference to FIG. 3. In a step SI, it is 
determined whether or not a user gives instructions of 
taking picture. When instructions of taking picture are 

10 given, a step S2 is performed. When no instructions of 
taking picture are given, such instructions are waited 
for by the system 1. At this occasion, as a result of a 
release button (manual button) being pressed by the user 
at a first viewpoint, the step S2 is performed, and a 

15 picture of the subject is taken. 

First, the above-mentioned pattern light is 
applied onto the subject by the pattern light 
irradiating part 14, and, then, a picture of the subject 
on which the pattern light is projected is taken. Then, 

20 a picture of the same subject but on which the pattern 
light is not applied is also taken. In the shape 
measurement system 1, it is also possible that 
previously obtained image data is input to the image 
memory 13 through the external terminal 5, instead of 

25 taking a picture of the subject. 
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Then, in a step S3, the attitude of the system 
1 at the occasion of taking the picture is detected, and 
three-dimensional coordinates of points present on the 
surface of the subject are calculated. How to detect 
5 the attitude will be described later in detail. As 
shown in FIG. 4, it is assumed that a picture of the 
subject 2 0 onto which the pattern light is projected by 
the pattern light irradiating part 14 is taken, and, 
thus, an image 21 is obtained. Then, a three- 
10 dimensional coordinate (Ax, Ay, Az) of a point A present 
on the subject 20 shown in FIG. 4 is calculated as 
follows : 

The above-mentioned three-dimensional 
coordinate (Ax, Ay, Ax) can be calculated by the 
15 following formula (1) according to the principle of 
triangulation : 



20 



/ J 

y / 



where : 
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(Px, Py, Pz) denotes a vector 23 representing 
a direction in which the pattern light is applied; 

(Ix, Iy) denotes a positional coordinate of 
the point A on the image 21 assuming that the point at 
5 which the optical axis of the lens 10 and the image 
surface intersect is the origin; 

f denotes the focal length of the picture 
taking system including the lens 10; and 

d denotes the distance between the pattern 
10 light irradiating part 14 and the optical center of the 
above-mentioned picture taking optical system. 

Then, a picture of the subject is taken again 
from a second viewpoint other than the above-mentioned 
first viewpoint similarly to the above-mentioned 
15 operation. Then, based on a thus-obtained image, the 
three-dimensional coordinates of points on the surface 
of the subject are calculated similarly to the above- 
mentioned operation. The operation of taking picture 
from the second viewpoint is performed in a manner such 
20 that at least some points on the surface of the subject 
taken from the first viewpoint are included in the image 
obtained in the operation of taking picture from the 
second viewpoint. 

Then, after a picture of the subject 20 is 
25 taken from a plurality of viewpoints as mentioned above, 



-25- 



a step S5 is performed for finishing taking picture of 
the same subject 20, but the step SI is performed again 
for continuing taking picture of the same subject 20. 

In the step S5 , the three-dimensional position 
5 composing part 17 combines three-dimensional coordinates 
(three-dimensional positional data) calculated for the 
above-mentioned respective viewpoints on a single 
coordinate system according to information concerning 
the corresponding attitudes detected by the attitude 

10 detecting part 16. A concrete example of the 

combination/ composing operation will be described later. 

Further, in the step S5, the image obtained 
through taking a picture of the subject on which the 
above-mentioned pattern light is not projected is 

15 combined with the three-dimensional shape obtained 

through the above-mentioned combination. Thereby, a 
three-dimensional image of the subject in which also an 
original pattern present on the surface of the subject 
is reproduced can be obtained. 

2 0 The operation in the step S3 performed by the 

attitude detecting part 16 will now be described in 
detail. First, as shown in FIG. 5, the attitude 
detecting part 16 includes a gravitation direction 
detecting part 16a including an acceleration sensor for 

25 measuring acceleration in each of three axes 



-26- 



perpendicular to each other, an around-gravitation angle 
detecting part 16b including a magnetic sensor for 
measuring a magnetic force in each of the above- 
mentioned three axes, and an attitude calculating part 
5 16c. The gravitation direction detecting part 16a is 

connected to the MPU 3 , and the around-gravitation angle 
detecting part 16b is connected to the gravitation 
direction detecting part 16a. The attitude calculating 
part 16c is connected to the gravitation direction 
10 detecting part 16a and around-gravitation angle 
detecting part 16b. 

The above-mentioned attitude detecting part 16 
may be made of a gyro. This case will be described 
later. 

15 First, an example in which the gravitation 

direction detecting part 16a is made of a three-axis 
acceleration sensor and the around-gravitation angle 
detecting part 16b is made of two-axis magnetic bearing 
sensor will now be described. In this example, as shown 

20 in FIG. 6, an XYZ basic coordinate system (absolute 
coordinate system) in which Y axis represents the 
gravitation direction, and a UVW picture taking 
coordinate system in which W axis represents the optical 
axis are defined. Then, attitude angles 9 0 , G v , G w 

25 denote inclinations of the picture taking coordinate 
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system with respect to the basic coordinate system, 

respectively . 

As a method of specifying the attitude of the 

shape measurement system 1, first, from a condition in 
5 which the basic coordinate system coincides with the 

picture taking coordinate system, the shape measurement 

system 1 is rotated by 9 V around the V (Y) axis. Then, 

with respect to the picture taking coordinate system, 

the shape measurement system 1 is rotated by 9 Q around 
10 the U axis. Then, further, with respect to the picture 

taking coordinate system, the shape measurement system 1 

is rotated by 0 W around the W axis. 

Assuming that R v , Ru, Rw denote rotation 

matrixes representing the above-mentioned rotations 
15 around the V axis, U axis and W axis, respectively, the 

attitude of the shape measurement system 1 can be 

expressed by the following matrix: 



R - R v - Ry • R vv 

" CosO Y 0 SinOy 

0 1 0 
- SinOy 0 CosQ Y 



1 0 
0 Cos9 x 
0 Sine y 



0 

- Sin9 x 
Cos9 y 



CosG z - Sin9 z 0 

SinB z CosQ z 0 

0 0 1 

• • • (2) 



Further, measured values obtained by the 
acceleration sensor and magnetic bearing sensor are 
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expressed by the following matrix A and matrix M, 
respectively: 



A Y 

\ A zJ 

My 

||A|| = |M|| = 1 



M 



However, when the two-axis magnetic bearing 
sensor is used, M Y is indefinite. The gravitational 
acceleration vector is expressed by the following matrix 
G: 



= 1 

,0, 



15 Further, assuming that the magnetic dip 

the terrestrial magnetism is already known, the 
terrestrial magnetism vector is expressed by the 
following matrix D: 
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0 
Sin(j> 
Cos<j)J 



The following formula holds between the above- 
5 mentioned matrix A and matrix G, and the matrix R to be 
obtained : 



R- A = G 

... (6) 

10 

Accordingly, by multiplying the inverse matrix 
of the matrix R from the left of both sides of the 
above-mentioned formula (6) , the following formula 
holds : 

15 

A = R*G 

• ' • (7) 



Thereby, the following attitude angles Gu and 
20 8 H are obtained through the attitude calculating part 
16c, shown in FIG. 5: 



6 U = —Sin l A z 
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4-AxJ\>o t(m d^L>0 
[Cosd v 

6 w =7Z-SinA Ax [ where: SinA A * \>0,and ^-<Q 



{Cos9 u J {CosB v 

-M-A—l wAere: im" 1 ! — ^— ] < 0, and 
\CosQ u ) \CosB v ) 9 V 

• • • (8 



Further, the following formula holds between 
5 the above-mentioned matrix M and matrix D, and the 
matrix R to be obtained: 



R • M = D 

... {9) 

10 

Accordingly, by multiplying the inverse matrix 
of the matrix R from the left of both sides of the 
above-mentioned formula (9) , the following formula 
holds : 

15 

M = R 1 D 

• • • (10) 



Thereby, the following formula (11) is derived, 
2 0 and, thus, the attitude angle 9 V is obtained through the 
attitude calculating part 16c, shown in FIG. 5: 
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CosQ Y = Sec<f> ■ Seedy ■ {M z + Sinfy ■ Sin0 v ) 

Sin6 v = -Seccj) ■ Secd w ■ Sin6 w ■ (M x - Siruff ■ Cos6 v - CosO v ■ Cos<j) ■ Study ) 

• * • (ID 



9 V = Sin' 1 (SinO v ) where : Sin' 1 (Sin6 v ) > 0, and CosG v > 0 

Qy = % - Sin' 1 (SinO v ) where : Sin" 1 (Sin6 v ) > 0, and Cosd v < 0 

6 V = -K - Sin' 1 [SinOy ) where : Sin~ l (Sin6 v ) < 0, and Cos6 v < 0 

5 • • • (12) 



When Cos 0u in the above-mentioned formula (8) 
is 0, the attitude angle 9 W can be determined freely. 

Through the above-described method, the matrix 
10 R shown in the formula (2) is obtained, and, thus, the 
information specifying the picture taking attitude of 
the shape measurement system 1 is obtained. 

An example in which a three-axis acceleration 
sensor and a three-axis magnetic bearing sensor are used 
15 will now be described. Definition of the coordinate 

systems and derivation of the attitude angles 0 Uf and 9 W 
are the same as those described above. However, the 
magnetic dip of the terrestrial magnetism may be un- 
known, and the terrestrial magnetism vector is expressed 
20 by the following matrix D: 
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D Y 



(13) 



The matrix D' shown in the following formula 
5 (14) is assumed: 



D' 
D' 



10 Then, by using the above-mentioned matrix D 1 

the above-mentioned matrix D is expressed by the 
following formula (15) : 



Thereby, the following formula (16) holds 
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Cos8 v = 



D' x 2 + D' z 2 



(16) 



Accordingly, the attitude angle 8 V is obtained 
5 as follows, through the attitude calculating part 16c: 



6 V = Sin' 1 (SinQy ) where : Sin' 1 (Sin0 v ) > 0, and CosQ v > 0 

d v =71- Sin' 1 (Sind v ) where : Sin~ l (Sind v ) > 0, and Cosd v < 0 

6 V = -% - Sin' 1 (Sind v ) where : Sin' 1 (Sin9 v ) < 0, and Cos9 v < 0 

' • ' (17) 



10 A case where a gyro is used as the attitude 

detecting part 16 shown in FIG. 1 will now be described. 
In this case, as will be described, a gyro is disposed 
so that a rotational angular velocity around each axis 
of the XYZ basic coordinate system (absolute coordinate 

15 system) shown in FIG. 6 can be obtained, and, by 

integrating the rotational angular velocity detected by 
the gyro, the angle information can be obtained. 

Specifically, assuming that a denotes the 
rotational angle around X axis, f3 denotes the rotational 

20 angle around Y axis, and y denotes the rotational angle 
around Z axis, shown in FIG. 6, the following rotational 
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angular velocity vector J is measured through the gyro: 



dt dt dt 



Then, by integrating the above-mentioned 
rotational angular velocity vector J, the rotational 
angles a, (3 and y at a time t can be obtained as 
follows : 



a 



a = a 0 + d 
P = /3 0 + [dp 

r = r o + \l d r 



Each of the above-mentioned integration 
15 coefficients a 0 , Po , Yo is 0 assuming that the initial 
state at the time t = 0 is a static state. 
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The composing operation for three-dimensional 
positional data in the step S5 shown in FIG. 3 will now 
be described in detail. FIGS. 7A through- 7D show the 
composing operation typically. As shown in FIGS . 7A 
5 through 7D, one example in which a three-dimensional 
shape of the subject 2 0 having a cylindrical shape is 
measured will now be described. 

FIG. 7B show a group of points 27 which 
represent three-dimensional coordinates corresponding to 

10 respective points present on the surface of the subject 
20 obtained from calculation in a predetermined 
coordinate system in a condition in which a picture of 
the subject 2 0 has been taken from a viewpoint 
(direction of picture taking) DB shown in FIG. 7A. 

15 Similarly, FIG. 7C show a group of points 28 which 

represent three-dimensional coordinates corresponding to 
respective points on the surface of the subject 2 0 
obtained from calculation in a predetermined coordinate 
system in a condition in which a picture of the subject 

20 20 has been taken from a viewpoint (direction of picture 
taking) DC shown in FIG. 7A. 

As shown in FIGS. 7B and 7C, depending on the 
direction of picture taking, a side surface, a rear 
surface, a top surface, or a bottom surface of the 

25 subject 20 becomes a blind spot for the picture taking 
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operation, for example. Thereby, it may not be possible 
to specify a shape of the subject 20 from each group of 
the thus-obtained groups of points 27 and 28 along. 
Accordingly, as shown in FIG. 7D, the groups of points 
5 27 and 28 shown in FIGS. 7B and 7C are combined in a 
same/single coordinate system. Thereby, a group of 
points 29 by which a three-dimensional shape of the 
subject 20 can be specified can be obtained. 

As the number of viewpoints from which a 

10 picture of a subject 20 is taken becomes larger, many 
groups of points will be obtained, and, thereby, it is 
possible to obtain a three-dimensional shape of the 
subject 20 more perfectly. 

The above-mentioned composing operation will 

15 now be described in more detail. Specifically, an 
example in which two sets of three-dimensional 
positional data obtained through taking picture of the 
subject 2 0 from the viewpoints DB and DC shown in FIG. 
7A will now be described. However, it is also possible 

20 to combine more than two sets of three-dimensional 
positional data. 

With regard to taking picture from the 
viewpoint DB , assuming that a rotational angle (Bx, By, 
Bz) around the XYZ axes shown in FIG. 6 from a basic 

25 attitude is given, a rotational matrix RB for coordinate 
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conversion from this basic attitude into the attitude of 
the viewpoint DB is expressed by the following formula 
(20) : 



RB= 



cosBycosBz -cosBxsinBz+sinBxsinBycosBz sinBxsmBz+cosBxsmBycosBz 
cosBysinBz cosSjccosBz+sinSjtsinB^sinfiz -sinBttosBz+cosBxsmBysinBz 
— sinBy sinBxcosBy cosBxzosBy 



(20) 



where it is assumed that the above-mentioned rotation is 
performed in the order of around X axis, around Y axis, 

10 and, then, around Z axis. 

Similarly, with regard to taking picture from 
the viewpoint DC, assuming that a rotational angle (Cx, 
Cy, Cz) around the XYZ axes shown in FIG. 6 from a basic 
attitude is given, a rotational matrix RC for coordinate 

15 conversion from this basic attitude into the attitude of 
the viewpoint DC is expressed by the following formula 
(21) : 



cos Cy cos Cz - cos Cx sin Cz + sin Cx sin Cy cos Cz sin Cx sin Cz + cos Cx sin Cy cos Cz 
cos Cy sin Cz cos Cx cos Cz + sin Cx sin Cy sin Cz — sin Cx cos Cz + cos Cx sin Cy sin Cz 
— sinCy sinCccosCy cosCxcosCy 



20 



(21) 
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Accordingly, a rotational matrix RR of 
converting a positional coordinate in the attitude 
(picture taking coordinate system) of the viewpoint DB 
into a positional coordinate in the attitude (picture 
5 taking coordinate system) of the viewpoint DC is 
expressed by the following: 

RR = (RB) _1 RC 

• ' ' (22) 

10 

Accordingly, a point having a positional 
coordinate b in the attitude (picture taking coordinate 
system) of the viewpoint DB is expressed by a positional 
coordinate obtained from RR • b in the attitude (picture 

15 taking coordinate system) of the viewpoint DC, for 

example. However, the above-mentioned matrix RR can 
perform coordinate conversion only for a rotational 
component. Accordingly, it is necessary to further 
perform conversion (shift) for a translation component. 

20 For this purpose, points corresponding to one 

another between the two images obtained from picture 
taking operation from the viewpoint DB and viewpoint DC 
are found out. For example, in both images shown in 
FIGS. 8A and 8B, points DDI and DD2 correspond to one 

25 another, and points DEI and DE2 correspond to one 
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another . 

For each point mentioned above, a three- 
dimensional coordinate in the picture taking coordinate 
system at the viewpoint DB or viewpoint DC is obtained. 
5 A positional difference in three-dimensional space 
obtained from removing a difference in rotational 
component through coordinate conversion by using the 
above-mentioned matrix RR performed on a three- 
dimensional coordinate in the picture taking coordinate 

10 system of the viewpoint DB is a required shift amount 

(translation movement amount). That is, assuming that b 
and c denote positional coordinates of corresponding 
points in the picture taking coordinate systems of the 
viewpoint DB and viewpoint DC, respectively, the above- 

15 mentioned shift amount s is obtained from 

s = (RR • b) - c 

The above-mentioned shift amount s for all the 
20 corresponding points should be the same vector. However, 
in many cases, it has a different value due to noise or 
as result of erroneous corresponding points being taken. 
Accordingly, an average of the above-mentioned shift 
amounts calculated for the respective corresponding 
25 points is used as the translation component between the 
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viewpoints DB and DC. Further, instead of such 
averaging, the shift amount calculated from 
corresponding points for which there is a low- 
possibility that a proper correspondence relationship is 
5 not obtained may be used as the translation component. 

Then, all the three-dimensional coordinates in 
one picture taking coordinate system are shifted by the 
thus-obtained shift amount, and, as mentioned above, the 
three-dimensional coordinates in both picture taking 

10 coordinate systems are combined in a single coordinate 
system. At this occasion, when corresponding points do 
not completely coincide with one another, the position 
of center of gravity between the corresponding points 
(mid point therebetween when the corresponding points 

15 are two points) is used as a final three-dimensional 

coordinate. Thereby, it is possible to obtain a three- 
dimensional shape of the subject at a high accuracy. 

The above-described shape measurement system 1 
can be made of a camera 4 and a personal computer PC as 

20 shown in FIG. 9. That is, as shown in FIG. 10, in the 
camera 4, all the parts other than the 3D position 
calculating part 15 and three-dimensional position 
composing part 17 shown in FIG. 1 are contained in a 
single housing, while, the 3D position calculating part 

25 17 and three-dimensional position composing part 17 are 
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embodied by the personal computer PC separate from the 
camera 4 . 

In the shape measurement system having this 
configuration, an image signal and information 
5 representing picture taking attitude are obtained by the 
camera 4 first, and are stored in the image memory 13. 
Then, the image signal and information representing the 
picture taking attitude stored in the image memory 13 
are recorded into a carriable recording medium such as a 

10 SmartMedia or the like. Thereby, by loading this 

carriable recording medium into the personal computer PC, 
these signal and information are taken by a 
memory/communication interface part 19. Instead of 
using the carriable recording medium as mentioned above, 

15 it is also possible that, through a communication device 
not shown in the figure, the signal and information are 
transmitted into the memory /communication interface part 
19 wirelessly. 

Thus, in the shape measurement system shown in 

20 FIGS. 9 and 10, according to the signal and information 
taken into the memory/ communication interface part 19, 
the personal computer PC calculates three-dimensional 
coordinates corresponding to respective points on the 
surface of a subject, and, thus, generates a three- 

2 5 dimensional image of the subject. 
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Further, in the shape measurement, system shown 
in FIG. 10, the pattern light irradiating part 14 and 
attitude detecting part 16 are controlled by the MPU 3 
included in the camera 4. However, instead thereof, it 
5 is also possible that, as shown in FIG. 11, they may be 
controlled by control signals provided by the personal 
computer PC . 

Further, it is also possible that the above- 
described functions/operations of the shape measurement 

10 system 1 may be embodied by a software program. That is, 
these functions/operations may be described by 
instructions of a computer program, and, as a result of 
this program being executed by the MPU 3, these 
functions/operations can be easily achieved. In this 

15 occasion, the program may be previously recorded into a 
recording medium such as a CD-ROM 171, or the like, 
shown in FIG. 12, then, as shown in the figure, this 
recording medium may be loaded into the personal 
computer PC, and, thus, the program is read by and 

2 0 executed by the MPU 3. 

Thus, according to the shape measurement 
system 1 in the first embodiment of the present 
invention, an attitude (viewpoint) at which a picture of 
a subject is taken is detected by the attitude detecting 

25 part 16, and, according to the thus-detected attitudes 
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and images obtained through the picture taking operation, 
a three-dimensional shape of the subject can be obtained. 
Accordingly, it is possible to provide a miniaturized 
and inexpensive shape measurement system. 
5 Further, through the camera 2 shown in FIG. 2, 

by taking pictures of a subject while moving the camera 
2 by hands, it is possible to obtain a three-dimensional 
shape of the subject through the camera 2. Accordingly, 
it is possible to easily measure a three-dimensional 
10 shape for any of variable types of subjects. 

FIG. 13 shows a block diagram of a shape 
measurement system in a second embodiment of the present 
invention . 

As shown in the figure, the shape measurement 
15 system 35 in the second embodiment has the same 

configuration as the shape measurement system 1 in the 
first embodiment shown in FIG. 1 except that an 
interpolation processing part 50 is added . 

The interpolation processing part 50 is 
20 controlled by the MPU 3, and is connected to the image 
memory 13 . 

The shape measurement system 3 5 having the 
configuration shown in FIG. 13 operates similarly to the 
shape measurement system 1 in the first embodiment shown 
25 in FIG. 1. However, the interpolation processing part 
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50 operates, as follows: 

In the above-mentioned three-dimensional 
positional data obtained through taking pictures of a 
subject, the data may not .necessarily be present in a 
5 required spatial resolution in a three-dimensional 

coordinate system. Therefore, in a case where a group 
of points 27 shown in FIG. 14A are obtained, for example 
interpolation points 37 may be added so as to result in 
a finer arrangement of points in the three-dimensional 

10 coordinate system, as shown in FIG. 14B. 

As a method of the above-mentioned 
interpolation, any conventional method such as that 
employing a spline function for obtaining an 
approximation curve connecting such a group of points, 

15 or the like. 

Further, the above-mentioned interpolation 
processing part 50 may also perform correction 
processing as will now be described. That is, when an 
erroneously processed point PF is found out which is 

20 obviously discrete from a group of points of a group of 

points included in an image 39 obtained, as shown in FIG 
15A, a user may specify this point PF by a pointer 
(indicated by an arrow in the figure) through mouse 
operation, and, then, move it to a desired position, or 

25 delete it. Thus, it is possible to correct the 
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erroneously processed point PF. Then, by performing 
this correction processing, it is possible to obtain a 
proper image 41 shown in FIG. 15B. 

It is preferable that correction processing 
5 such as that described above is performed before the 
above-mentioned interpolation processing is performed. 

Further, it is possible that the above- 
mentioned interpolation processing part 50 is contained 
in a housing together with the other parts/components, 

10 and, thus, a camera is made. Alternatively, it is also 
possible that, as shown in FIG. 16, the interpolation 
processing part' 50 is provided between the 3D position 
calculating part 15 and three-dimensional position 
composing part 17 in the personal computer PC. 

15 Thus, according to the shape measurement 

system 35 in the second embodiment, advantages which are 
obtained from the above-mentioned shape measurement 
system 1 in the first embodiment are obtained, and, also, 
three-dimensional positional data is interpolated or 

20 corrected by the interpolation processing part 50, and, 
thereby, it is possible to obtain a three-dimensional 
shape of a subject at a higher accuracy. 

Furthermore, according to the shape 
measurement system 35 in the second embodiment, as 

25 interpolation such as that mentioned above can be made, 
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it is possible to employ the pattern light irradiating 
part 14 for projecting a predetermined pattern onto a 
subject having a simplified configuration. Accordingly, 
it is possible to miniaturize the shape measurement 
5 system. 

Further, according to the shape measurement 
system 35 in the second embodiment, as interpolation 
such as that mentioned above can be made, it is possible 
to avoid influence of an unexpected movement of the 

10 system 35 when a picture of a subject is taken through 
the system 35 while it is held by hands, or the like. 
Thereby, it is possible to obtain a three-dimensional 
shape of the subject at a higher accuracy. 

Same as for the shape measurement system 1 in 

15 the first embodiment, the above-described 

functions/operations of the shape measurement system 35 
in the second embodiment may be embodied by a software 
program . 

FIG. 17 shows a block diagram of a shape 
20 measurement system in a third embodiment of the present 
invention . 

As shown in FIG. 17, the shape measurement 
system 45 in the third embodiment has the same 
configuration as that of the shape measurement system 1 
25 in the first embodiment except that a lens 10a, a stop 
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mechanism 6a, a picture taking device 11a, a CDS circuit 
7a, an A-D converter 12a, a timing signal generating 
circuit 8a, and an IPP circuit 9a are added. The stop 
mechanism 6a, timing signal generating circuit 8a and 
5 IPP circuit 9a are controlled by the MPU 3. The picture 
taking device 11a, CDS circuit 7a and A-D converter 12a 
operate according to signals provided by the timing 
signal generating circuit 8a. The IPP circuit 9a is 
connected to the image memory 18. 

10 In the shape measurement system 45 in the 

third embodiment, the two picture taking systems 
including lenses, CCDs and so forth, respectively, are 
provided in a manner such that the optical centers 
thereof are different from one another. The difference 

15 between the optical centers, difference between optical 
axes in angle and so forth between these two picture 
taking systems are previously obtained precisely. 

The shape measurement system 45 in the third 
embodiment having the above-described configuration 

20 operates similarly to the above-mentioned shape 

measurement system 1 in the first embodiment. However, 
through one picture taking operation performed on a 
subject, two images of the subject can be obtained 
through the two lenses 10 and 10a. Thereby, it is 

25 possible to obtain two sets of three-dimensional 
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positional data through one picture taking operation. 

Accordingly, according to the shape 
measurement system 45 in the third embodiment, by 
combining thus-obtained all the plurality of sets of 
5 three-dimensional positional data into a single 
coordinate system, it is possible to improve the 
accuracy of a finally obtained three-dimensional shape 
of the subject. 

Although the above-described third embodiment 
10 includes the two picture taking systems, the present 

invention includes a shape measurement system in another 
embodiment which includes more than two picture taking 
systems . 

Same as for the shape measurement systems 1 
15 and 35 in the first and second embodiments, the above- 
described functions/operations of the shape measurement 
system 45 in the third embodiment may also be embodied 
by a software program. 

Further, the present invention is not limited 
2 0 to the above-described embodiments, and variations and 
modifications may be made without departing from the 
scope of the present invention. 

The present application is based on Japanese 
priority application No. 2000-240470, filed on August 8, 
25 2000, the entire contents of which are hereby 
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incorporated by reference . 



